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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address- 
Mi claims being allowable, PROSECUTION ON THE MERITS IS (OR REIVIAINS) CLOSED in this application. If not included 
herewith (or previously mailed), a Notice of Allowance (PTOL-85) or other appropriate communication wilt be mailed in due course. THIS 
NOTICE OF ALLOWABILITY IS NOT A GRANT OF PATENT RIGHTS. This application is subject to withdrawal from issue at the initiative 
of the Office or upon petition by the applicant. See 37 CFR 1.313 and MPEP 1 308. 

1 . 13 This communication is responsive to communication on 08/30/06. . 

2. K The allowed claim(s) is/are U5. 7-9. lUld AND RENUMBERED AS 1-16 . 

3. □ Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 

a) □ All b) □ Some* c) □ None of the: 

1 . □ Certified copies of the priority documents have been received. 

2. □ Certified copies of the priority documents have been received in Application No. . 

3. □ Copies of the certified copies of the priority documents have been received in this national stage application from the 

International Bureau (PCT Rule 17.2(a)). 
* Certified copies not received: . 

Applicant has THREE MONTHS FROM THE "MAILING DATE" of this communication to file a reply complying with the requirements 
noted below. Failure to timely comply will result in ABANDONMENT of this application. 
THIS THREE-MONTH PERIOD IS NOT EXTENDABLE. 

4. □ A SUBSTITUTE OATH OR DECLARATION must be submitted. Note the attached EXAMINER'S AMENDMENT or NOTICE OF 

INFORMAL PATENT APPLICATION (PTO-152) which gives reason(s) why the oath or declaration Is deficient. 

5. □ CORRECTED DRAWINGS ( as "replacement sheets") must be submitted. 

(a) □ including changes required by the Notice of Draflsperson's Patent Drawing Review ( PTO-948) attached 

1 ) □ hereto or 2) □ to Paper No./Mail Date . 

(b) □ including changes required by the attached Examiner's Amendment / Comment or in the Office action of 

Paper No./Mail Date . 

Identifying indicia such as the application number (see 37 CFR 1.84(c)) should be written on the drawings in the front (not the back) of 
each sheet. Replacement sheet(s) should be labeled as such in the header according to 37 CFR 1.121(d). 

6. □ DEPOSIT OF and/or INFORMATION about the deposit of BIOLOGICAL MATERIAL must be submitted. Note the 

attached Examiner's comment regarding REQUIREMENT FOR THE DEPOSIT OF BIOLOGICAL MATERIAL 
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DETAILED ACTION 

1 . This Office Action is Response to the phone interview with Brian Dingman on August 
30, 2006. 

2. The Double Patenting Rejection in the previous Office Action mailed on March 09, 2005 
is withdrawn due to the Terminal Disclaimer filed on May 25, 2006. 

EXAMINER'S AMENDMENT 

3. An examiner's amendment to the record appears below. Should the changes and/or 
additions be unacceptable to appUcant, an amendment may be filed as provided by 37 CFR 
1.312. To ensure consideration of such an amendment, it MUST be submitted no later than the 
payment of the issue fee. 

Authorization for this examiner's amendment was given in a telephone interview with 
Brian Dingman on August 30, 2006. 

The application has been amended as follows: 
In the claims: 

1 . (Currently Amended) A method for automatically tracking an object to keep the object in 
the field of view of a video camera, and create an augmented reality display comprising the 
image fi^om the video camera combined with computer-generated graphics to create an 
augmented reahty display, the method comprising: 
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providing a motorized camera mount for the camera, and controlling the camera mount 
using a computer: 

resolving the camera's field of view; 

determining the location of an object to be tracked relative to the camera's field of view; 

in response to the determined location of the object, received by the computer from the 
motorized camera mount , moving the camera to maintain the object in the camera's field of view; 

wherein the resolving step comprises using the computer to resolve the field of view 
based on the current camera position received bv the computer from the camera mount, to 
accomplish a feedback control svstem: 

using [[a]] the computer to generate a graphical image representing unseen information 
that corresponds to the camera's viewpoint; and 

augmenting the video image with the graphical image, to create an augmented-reality 
image having the tracked object therein, for presentation to a user. 

2. (Original) The method of claim 1 wherein the augmenting step comprises using 
onboard video mixing through use of a video capture device with the computer. 

3. (Original) The method of claim 1 wherein the augmenting step comprises using an 
external video mixing solution, to combine real and computer-generated graphical elements 
outside of the computer. 

4. (Original) The method of claim 1 for use in operations. 

5. (Original) The method of claim 1 for use in training. 

6. (Cancelled) 
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7. (Currently Amended) The method of claim [[6]] i in which the resolving step 
comprises calibrating the camera and camera mount. 

8. (Currently Amended) The method of claim [[6]] i in which the camera mount is 
coupled to a fixed platform. 

9. (Currently Amended) The method of claim [[6]] in which the resolving step comprises 
using the camera and camera mount in conjunction with a separate camera position tracking 
system to generate a combined position and orientation value. 

10. (Cancelled) 

1 1 . (Currently Amended) The method of claim [[6]] i in which the computer controls the 
camera mount, and the resolving step comprises using the computer to position the camera in a 
feed-forward control system. 

12. (Currently Amended) The method of claim [[6]] i in which the camera mount is not 
stationary. 

13. (Original) The method of claim 12 in which the camera mount is attached to a 

vehicle. 

14. (Original) The method of claim 12 in which the camera mount is attached to an 

aircraft. 

15. (Original) The method of claim 12 in which the camera mount is attached to a 
watercraft or ship. 

16. (Original) The method of claim 12 in which the camera mount is attached to a 
gimbaled arm. 
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17. (Currently Amended) The method of claim [[6]] i in which the resolving step 
comprises the motorized camera mount reporting the field of view of the camera to the computer. 

18. (Currently Amended) A method for automatically tracking an object to keep the 
object in the field of view of a video camera, and create an augmented reality display comprising 
the image fi-om the video camera combined with computer-generated graphics to create an 
augmented reality display, the method comprising: 

providing a motorized camera mount for the camera; 
resolving the camera's field of view; 

determining the location of an object to be tracked relative to the camera's field of view; 

in response to the determined location of the object, moving the camera to maintain the 
object in the camera's field of view; 

using a computer to control the camera mount, and wherein the resolving step comprises 
using the computer to resolve the field of view based on the current camera position received bv 
the computer firom the camera mount, to accomplish a feedback control svstem: 

using [[a]] tiie computer to generate a graphical image representing unseen information 
that corresponds to the camera's viewpoint; 

using the computer to control the camera's viewpoint; and 

using a video capture device along with the computer to accomplish video mixing that 
augments the video image with the graphical image, to create an augmented-reality image having 
the tracked object therein, for presentation to a user. 
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Allowable Subject Matter 

4. Claims 1-5, 7-9 and 11-18 allowed and renumbered as 

5. The following is an examiner's statement of reasons for allowance: The present invention 
relates to a method of automatically tracking an object to keep the object in the field of view of 
the of a video camera, and create and augmented reahty display comprising the image fi'om the 
video camera combined with the computer generated graphics to create an augmented reality 
display. 

Each independent claim identifies the uniquely distinct features "providing a motorized 
camera mount for the camera; and using a computer to control the camera mount, in response to 
the determined location of the object, received by the computer from the motorized camera 
mount, moving the camera to maintain the object in the camera's field of view; and wherein the 
resolving step comprises using the computer to resolve the field of view based on the current 
camera position received by the computer fi-om the camera mount, to accomplish a feedback 
control system". 

The closest prior art of AUenby et al, (U.S Patent No. 5,815,41 1) and Massunaga et al. 
(U.S Patent No. 5,838,368) show similar systems, but either singularly or in combination, fail to 
anticipate or render above quoted limitations obvious. 

6. Any comments considered necessary by applicant must be submitted no later than the 
payment of the issue fee and, to avoid processing delays, should preferably accompany the issue 
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fee. Such submissions should be clearly labeled "Comments on Statement of Reasons for 
Allowance." 



7. Any inquiry concerning this conununication or earlier communications from the 
examiner should be directed to DUG Q. DINH whose telephone number is (571) 272-7686. The 
examiner can normally be reached on Mon-Fri from 8:00.AM-4:00.PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, RICHARD HJERPE can be reached on (571) 272-7691. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 
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